Trước hết theo BĐT Schur bậc 3 ta có:

\(\left(a+b+c\right)\left(a^2+b^2+c^2\right)+9abc\ge2\left(a+b+c\right)\left(ab+bc+ca\right)\)

\(\Leftrightarrow a^2+b^2+c^2+3abc\ge2\left(ab+bc+ca\right)\) (do \(a+b+c=3\)) (1)

Đặt vế trái BĐT cần chứng minh là P, ta có:

\(P=\dfrac{\left(a^2+abc\right)^2}{a^2b^2+2abc^2}+\dfrac{\left(b^2+abc\right)^2}{b^2c^2+2a^2bc}+\dfrac{\left(c^2+abc\right)^2}{a^2c^2+2ab^2c}\)

\(\Rightarrow P\ge\dfrac{\left(a^2+b^2+c^2+3abc\right)^2}{a^2b^2+b^2c^2+c^2a^2+2abc\left(a+b+c\right)}=\dfrac{\left(a^2+b^2+c^2+3abc\right)^2}{\left(ab+bc+ca\right)^2}\)

Áp dụng (1):

\(\Rightarrow P\ge\dfrac{\left[2\left(ab+bc+ca\right)\right]^2}{\left(ab+bc+ca\right)^2}=4\) (đpcm)

Dấu "=" xảy ra khi \(a=b=c=1\)

Hi vọng là tìm GTLN:

Không mất tính tổng quát, giả sử b, c cùng phía với 1 \(\Rightarrow\left(b-1\right)\left(c-1\right)\ge0\Leftrightarrow bc\ge b+c-1\).

Áp dụng bất đẳng thức AM - GM ta có: 

\(4=a^2+b^2+c^2+abc\ge a^2+2bc+abc\Leftrightarrow2bc+abc\le4-a^2\Leftrightarrow bc\left(a+2\right)\le\left(2-a\right)\left(a+2\right)\Leftrightarrow bc+a\le2\)

\(\Rightarrow a+b+c\le3\).

Áp dụng bất đẳng thức Schwarz ta có:

\(P\le\dfrac{ab}{9}\left(\dfrac{1}{a}+\dfrac{2}{b}\right)+\dfrac{bc}{9}\left(\dfrac{1}{b}+\dfrac{2}{c}\right)+\dfrac{ca}{9}\left(\dfrac{1}{c}+\dfrac{2}{a}\right)=\dfrac{1}{9}.3\left(a+b+c\right)=\dfrac{1}{3}\left(a+b+c\right)\le1\).

Đẳng thức xảy ra khi a = b = c = 1.

Ta có: P= \(2a+3b+\dfrac{1}{a}+\dfrac{4}{b}\) = \(\text{​​}\text{​​}(\dfrac{1}{a}+a)+\left(\dfrac{4}{b}+b\right)+\left(a+2b\right)\)

Ta thấy: \(\text{​​}\text{​​}(\dfrac{1}{a}+a)\ge2\sqrt{\dfrac{1}{a}\cdot a}=2\)

             \(\text{​​}\text{​​}\left(\dfrac{4}{b}+b\right)\ge2\sqrt{\dfrac{4}{b}\cdot b}=4\)

Do đó: P \(\ge2+4+5=11\)

Vậy: P(min)=11  khi:  \(\left\{{}\begin{matrix}\dfrac{1}{a}=a\\\dfrac{4}{b}=b\end{matrix}\right.\Leftrightarrow\left\{{}\begin{matrix}a=1\\b=2\end{matrix}\right..\)

Đặt \(\left(\sqrt{a};\sqrt{b};\sqrt{c}\right)=\left(x;y;z\right)\Rightarrow x+y+z=1\)

BĐT trở thành: \(\dfrac{xy}{\sqrt{x^2+y^2+2z^2}}+\dfrac{yz}{\sqrt{y^2+z^2+2x^2}}+\dfrac{zx}{\sqrt{x^2+z^2+2y^2}}\le\dfrac{1}{2}\)

Ta có:

\(x^2+z^2+y^2+z^2\ge\dfrac{1}{2}\left(x+z\right)^2+\dfrac{1}{2}\left(y+z\right)^2\ge\left(x+z\right)\left(y+z\right)\)

\(\Rightarrow\dfrac{xy}{\sqrt{x^2+y^2+2z^2}}\le\dfrac{xy}{\sqrt{\left(x+z\right)\left(y+z\right)}}\le\dfrac{1}{2}\left(\dfrac{xy}{x+z}+\dfrac{xy}{y+z}\right)\)

Tương tự: \(\dfrac{yz}{\sqrt{y^2+z^2+2x^2}}\le\dfrac{1}{2}\left(\dfrac{yz}{x+y}+\dfrac{yz}{x+z}\right)\)

\(\dfrac{zx}{\sqrt{z^2+x^2+2y^2}}\le\dfrac{1}{2}\left(\dfrac{zx}{x+y}+\dfrac{zx}{y+z}\right)\)

Cộng vế với vế:

\(VT\le\dfrac{1}{2}\left(\dfrac{zx+yz}{x+y}+\dfrac{xy+zx}{y+z}+\dfrac{yz+xy}{z+x}\right)=\dfrac{1}{2}\left(x+y+z\right)=\dfrac{1}{2}\) (đpcm)

Dấu "=" xảy ra khi \(x=y=z\) hay \(a=b=c\)

\(\dfrac{ab}{\sqrt{ab+2c}}=\dfrac{ab}{\sqrt{ab+\left(a+b+c\right)c}}=\dfrac{ab}{\sqrt{\left(a+c\right)\left(b+c\right)}}=ab\cdot\sqrt{\dfrac{1}{a+b}\cdot\dfrac{1}{b+c}}\le ab\cdot\dfrac{1}{2}\left(\dfrac{1}{a+b}+\dfrac{1}{b+c}\right)=\dfrac{1}{2}\left(\dfrac{ab}{a+b}+\dfrac{ab}{b+c}\right)\)

CMTT: \(\dfrac{bc}{\sqrt{bc+2a}}\le\dfrac{1}{2}\left(\dfrac{bc}{a+b}+\dfrac{bc}{a+c}\right);\dfrac{ac}{\sqrt{ac+2b}}\le\dfrac{1}{2}\left(\dfrac{ac}{b+c}+\dfrac{ac}{b+a}\right)\)

\(\Leftrightarrow P\le\dfrac{1}{2}\left(\dfrac{ab}{c+a}+\dfrac{ab}{c+b}+\dfrac{bc}{b+a}+\dfrac{bc}{c+a}+\dfrac{ac}{b+c}+\dfrac{ac}{b+c}\right)\\ \Leftrightarrow P\le\dfrac{1}{2}\left[\dfrac{b\left(a+c\right)}{a+c}+\dfrac{a\left(b+c\right)}{b+c}+\dfrac{c\left(a+b\right)}{a+b}\right]=\dfrac{1}{2}\left(a+b+c\right)=1\)

Dấu \("="\Leftrightarrow a=b=c=\dfrac{2}{3}\)

\(\dfrac{ab}{\sqrt{ab+2c}}=\dfrac{ab}{\sqrt{ab+c\left(a+b+c\right)}}=\dfrac{ab}{\sqrt{\left(a+c\right)\left(b+c\right)}}\le\dfrac{1}{2}\left(\dfrac{ab}{a+c}+\dfrac{ab}{b+c}\right)\)

Tương tự:

\(\dfrac{bc}{\sqrt{bc+2a}}\le\dfrac{1}{2}\left(\dfrac{bc}{a+b}+\dfrac{bc}{a+c}\right)\) ; \(\dfrac{ca}{\sqrt{ac+2b}}\le\dfrac{1}{2}\left(\dfrac{ca}{a+b}+\dfrac{ca}{b+c}\right)\)

Cộng vế:

\(P\le\dfrac{1}{2}\left(\dfrac{bc}{a+b}+\dfrac{ca}{a+b}+\dfrac{ab}{a+c}+\dfrac{bc}{a+c}+\dfrac{ab}{b+c}+\dfrac{ca}{b+c}\right)=\dfrac{1}{2}\left(a+b+c\right)=1\)

\(P_{max}=1\) khi \(a=b=c=\dfrac{2}{3}\)

\(\sqrt{\dfrac{ab+2c^2}{1+ab-c^2}}=\sqrt{\dfrac{ab+2c^2}{a^2+b^2+ab}}\)\(=\dfrac{ab+2c^2}{\sqrt{\left(a^2+b^2+ab\right)\left(ab+c^2+c^2\right)}}\)\(\ge\dfrac{2\left(ab+2c^2\right)}{a^2+b^2+2ab+2c^2}\)\(\ge\dfrac{2\left(ab+2c^2\right)}{2\left(a^2+b^2\right)+2c^2}\)\(=\dfrac{ab+2c^2}{a^2+b^2+c^2}\)

\(\Rightarrow\sqrt{\dfrac{ab+2c^2}{1+ab-c^2}}\ge ab+2c^2\)

Tương tự: \(\sqrt{\dfrac{bc+2a^2}{1+bc-a^2}}\ge bc+2a^2\); \(\sqrt{\dfrac{ac+2b^2}{1+ac-b^2}}\ge ac+2b^2\)

Cộng vế với vế \(\Rightarrow VT\ge2a^2+2b^2+2c^2+ab+bc+ac=2+ab+bc+ac\)

Dấu = xảy ra khi \(a=b=c=\dfrac{1}{\sqrt{3}}\)

Đặt \(\left(a;b;c\right)=\left(\dfrac{y}{x};\dfrac{z}{y};\dfrac{x}{z}\right)\)

\(\Rightarrow VT=\dfrac{1}{\dfrac{y}{x}\left(\dfrac{z}{y}+1\right)}+\dfrac{1}{\dfrac{z}{y}\left(\dfrac{x}{z}+1\right)}+\dfrac{1}{\dfrac{x}{z}\left(\dfrac{y}{x}+1\right)}\)

\(VT=\dfrac{x}{y+z}+\dfrac{y}{z+x}+\dfrac{z}{x+y}=\dfrac{x^2}{xy+xz}+\dfrac{y^2}{xy+yz}+\dfrac{z^2}{xz+yz}\)

\(VT\ge\dfrac{\left(x+y+z\right)^2}{2\left(xy+yz+zx\right)}\ge\dfrac{3\left(xy+yz+zx\right)}{2\left(xy+yz+zx\right)}=\dfrac{3}{2}\)

\(abc\ge\left(a+b-c\right)\left(b+c-a\right)\left(c+a-b\right)\)

\(\Leftrightarrow abc\ge\left(3-2a\right)\left(3-2b\right)\left(3-2c\right)\)

\(\Leftrightarrow9abc\ge12\left(ab+bc+ca\right)-27\)

\(\Rightarrow abc\ge\dfrac{4}{3}\left(ab+bc+ca\right)-3\)

\(P\ge\dfrac{9}{a\left(b^2+bc+c^2\right)+b\left(c^2+ca+a^2\right)+c\left(a^2+ab+b^2\right)}+\dfrac{abc}{ab+bc+ca}=\dfrac{9}{\left(ab+bc+ca\right)\left(a+b+c\right)}+\dfrac{abc}{ab+bc+ca}\)

\(\Rightarrow P\ge\dfrac{3}{ab+bc+ca}+\dfrac{abc}{ab+bc+ca}=\dfrac{3+abc}{ab+bc+ca}\)

\(\Rightarrow P\ge\dfrac{3+\dfrac{4}{3}\left(ab+bc+ca\right)-3}{ab+bc+ca}=\dfrac{4}{3}\)

Dấu "=" xảy ra khi \(a=b=c=1\)